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THE  PROBLEM 

To  determine  if  target  identification  on  a  sonar  waterfall  display  is 
improved  when  target  information  is  presented  both  visually  and  auditorially 
compared  to  either  of  these  alone. 


THE  FINDINGS 

Target  identifications  were  made  about  1  dB  lower  with  bimodal  than 
unimodal  presentation  of  the  target  information. 


APPLICATION 

These  findings  support  the  use  of  bimodal  presentation  of  information 
to  operators  of  sonar,  radar,  ECM  equipment,  and  the  like. 
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ABSTRACT 


Identification  thresholds  were  measured  for  four  targets  which  were 
presented  on  a  CRT  display  or  through  earphones  under  5  conditions: 

(1)  auditory  target  with  auditory  noise;  (2)  visual  target  with  visual 
noise;  (3)  auditory  target  with  both  visual  and  auditory  noise;  (4)  visual 
target  with  both  visual  and  auditory  noise;  and  (5)  both  visual  and 
auditory  targets  with  both  visual  and  auditory  noise. 

Both  visual  and  auditory  thresholds  were  better  when  the  auditory  and 
visual  targets  were  presented  together  than  when  either  was  presented 
alone.  This  was  true  whether  the  two  stimuli  were  being  increased  in 
intensity  simultaneously  or  whether  the  secondary  stimuli  was  set  at  a 
constant  level  1  to  4  dB  below  the  baseline  threshold. 


THE  EFFECTS  OF  BIMCDAL  PRESENTATION  OF  STIMULI  AND  NOISE 

ON  TARGET  IDENTIFICATION 


Several  studies  have 
investigated  the  effects  on  an 
observer's  ability  to  detect  a 
stimulus  in  one  modality  when 
information  has  been  presented 
simultaneously  to  a  second 
modality.  But,  as  Kobus,  et.  al. 
(1985)  have  pointed  out,  most 
studies  of  the  interaction  of  two 
sensory  modes  have  used  such  simple 
stimuli  as  flashes  of  light, 
clicks,  and  tones  (Nickerson, 

1983) ;  moreover,  they  tend  to  use  a 
method  of  directed  or  divided 
attention  in  which  the  subject  must 
attend  to  one  mcdality  (e.g., 
vision)  while  another  modality 
(e.g. ,  audition)  competes  for 
attent ion . 

The  typical  findings  of  such 
studies  are  that  the  bimodal 
presentation  does  not  degrade 
performance  (Colquhoun,  1975) . 
Indeed,  several  studies  have  found 
that  reaction  time  and  signal 
detection  are  improved  with  bimodal 
stimulation  (see  Kobus,  et.  al, 

1985) .  Apparently,  individuals  can 
divide  their  attention  between  two 
modalities  quite  effectively 
(Loveless,  et.  al.,  1970;  Keele  and 
Neill,  1976),  and  information  from 
one  modality  can  enhance 
performance  based  on  another 
modality  (Hanson,  1981) . 

Few  such  studies  have  been  done 
with  complex  tasks.  Vet  it  is 
common  for  individuals  to  have  to 
deal  with  information  in  more  than 
one  modality  in  most  situations. 
Sonarmen  routinely  do  so.  Is  their 
performance  improved  if  target 
information  is  presented 
redundantly  both  visually  and 
auditorially? 


Kobus,  et.  al.  (1985)  and 
Lewandowski  and  Kobus  (1985) 
studied  the  effects  of  bimodal 
presentation  on  the  detection  of 
sonar  targets.  They  found  that 
these  thresholds  were  improved  by 
bimodal  stimulation.  The  present 
study  investigated  the  effects  of 
redundant  bimodal  stimulation  on 
the  more  difficult  task  of 
classification,  that  is, 
identifying  at  near-threshold 
levels  which  of  several  stimuli  is 
being  presented. 

EXPERIMENT  1 
METHOD 

Subjects.  Three  staff  members  of 
the  laboratory  and  one  sonarman 
volunteered  to  serve  as  subjects. 

Apparatus.  Three  visual  and 
auditory  frequencies,  300,  1000, 
and  1600  Hz,  were  produced,  each  by 
a  Wavetek  function  generator  (Model 
FG  501) .  They  were  presented  on 
both  a  monochromatic  visual  display- 
unit  (VDU)  and  Koss  ( PRO/ 4 AM ) 
headphones.  The  signal  from  each 
generator  was  split  into  two 
channels  and  fed  through  separate 
Hewlett-Packard  attenuators  (Model 
3 50D)  before  being  displayed. 

White  noise  was  generated  by  a 
Grason-Stadler  Noise  Generator 
(Model  901B) ,  also  split  into  two 
channels,  and  routed  through 
separate  attenuators  before  going 
to  the  VDU  and  the  headphones. 

The  VDU  displayed  the  signal 
frequency  along  thr>  x-axir-  and  time 
along  the  y-axis.  A  horizontal 
line  of  pixels  appeared  at  the  top 
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of  the  display  and  moved  down  the 
screen  in  a  "waterfall"  fashion. 

Each  line  remained  visible  for  6.2 
sec,  and  16  lines  were  present  at  a 
time.  The  update  time  was  thus  0.4 
sec,  equivalent  to  short-time 
averaging  on  these  sonar  displays. 

A  signal  appeared  as  an 
intermittent  vertical  line  of  dots 
of  greater  intensity  than  the 
background  noise.  The  1000  Hz 
signal  appeared  near  the  left  and 
right  edges  of  the  screen,  since 
the  range  of  the  display  was  from  0 
to  2000  Hz.  Intensity  was  coded  as 
eight  brightness  levels.  Noise 
appeared  as  lighted  pixels  varying 
randomly  in  position  and  intensity. 

A  target  consisted  of  any  two 
or  all  three  frequencies  presented 
together.  There  were  thus  four 
targets. 

Procedure.  Three  categories  of 
thresholds  were  measured.  In  the 
baseline  condition,  only  one 
sensory  modality  was  stimulated; 
target  information  and  noise  were 
both  presented  either  as  a  visual 
display  or  through  earphones.  In 
the  unimodal  condition,  the  noise 
was  presented  both  visually  and 
aurally,  but  the  target  was 
presented  in  only  one  of  the 
modalities,  and  separate  visual  and 
auditory  thresholds  were  again 
obtained.  In  the  bimodal 
condition,  the  same  target 
information  and  noise  was  presented 
to  both  modalities  at  the  same 
time. 

Subjects  were  thus  tested  under 
five  conditions;  (l)  auditory 
target  with  auditory  noise 
(auditory  baseline) ;  (2)  visual 
target  with  visual  noise  (visual 
baseline) ;  (3)  auditory  target  with 
both  auditory  and  visual  noise 


(auditory  unimodal) ;  (4)  visual 
target  with  visual  and  auditory 
noise  (visual  unimodal) ;  and  (5) 
both  bisual  and  auditory  target 
with  both  visual  and  auditory  noise 
(bimodal) . 

The  subjects  first  were  given 
an  extended  training  procedure 
during  which  they  learned  to 
identify  the  four  targets.  That 
is,  the  subject  was  required  to 
state  whether  the  target  consisted 
of  all  three  frequencies,  or  the 
high  and  low  frequencies,  etc. 

This  proved  to  be  rather  difficult 
with  the  auditory  targets  for  all 
the  subjects  except  one,  and 
training  required  about  half  a 
dozen  sessions.  The  experiment 
began  when  the  subject  could 
identify  each  of  the  four  targets, 
presented  randomly,  three  times 
with  no  more  than  one  error  out  of 
the  12  presentations. 

Every  session  began  with 
practice  in  identifying  the  four 
targets  followed  by  the  test 
procedure  to  ensure  that  they  could 
identify  the  targets.  Next,  the 
baseline  auditory  and  visual 
thresholds  were  measured  twice  for 
each  target.  After  this,  the 
unimodal  and  bimodal  conditions 
were  presented  randomly, 
intermixed,  until  one  threshold  had 
been  obtained  for  each  target  in 
each  condition. 

To  obtain  a  baseline  or  a 
unimodal  threshold,  the  target  was 
simply  presented  below  threshold 
and  its  attenuation  decreased  (that 
is,  the  intensity  increased)  by  1 
dB  every  10  sec  until  the  subject 
identified  the  target. 

To  obtain  a  bimodal  threshold, 
both  the  visual  and  auditory 
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targets  were  initially  presented  5 
dB  below  their  mean  baseline 
thresholds  for  that  session,  and 
their  attenuation  was  decreased  by 
1  dB  every  10  sec  until  the  subject 
identified  the  target  and  reported 
the  modality  in  which  the  target 
appeared.  In  most  cases,  only  one 
modality  was  reported  .ut  there 
were  instances  in 
observer  said  tha 
identified  the  tan. 
modalities.  In  eit 
thresholds  were  comp 
unimodal  ones. 


"he 

both 

ise,  these 
with  the 


At  the  end  of  the  session,  a 
few  baseline  thresholds  were  again 
measured  to  ensure  that  no 
significant  threshold  shifts  had 
occurred.  A  session  lasted  about 
45  minutes. 


There  were  five  sessions  for 
each  subject,  resulting  in  five 
thresholds  for  each  target  in  each 
condition. 


RESULTS 

Table  1  gives  the  mean 
attenuation  in  dB  at  which  the  four 
targets  were  identified  in  the 
various  conditions.  The  higher  the 
dB  level,  of  course,  the  greater 
the  observers'  sensitivity.  There 
was  little  change  in  the  thresholds 
for  the  targets  displayed  visually. 
The  baseline  and  unimodal 
thresholds  were  virtually 
identical;  the  bimodal  threshold 
was  improved  by  about  half  a  dB, 
but  none  of  these  changes  was 
significant,  according  to  repeated 
measures  analysis  of  variance 
(F ( 2 , 6) =1 . 60) .  The  only 
statistically  significant 
difference  among  the  visual 
thresholds  was  that  the  threshold 
for  the  target  composed  cf  the 


intermediate  and  high  frequency  was 
better  than  the  threshold  for  the 
target  composed  of  all  three 
frequencies . 

The  mean  auditory  threshold, 
however,  was  improved  by  nearly 
three  dB  in  th^  bimodal  condition, 
a  change  which  was  significant, 
according  to  the  analysis  of 
variance  (F(2 , 6) =21. 26,  p  <  .01). 
The  bimodal  threshold  was 
significantly  better  (p  <  .01)  than 
both  the  baseline  and  unimodal 
thresholds,  according  to  the 
Newman-Keuls  test. 

EXPERIMENT  2 

The  presentation  of  target 
information  visually  and 
auditorially  at  the  same  time 
clearly  improved  the  subjects' 
ability  to  make  an  auditory 
identification  of  the  targets,  but 
it  had  very  little  effect  on  the 
identification  of  the  visual 
targets.  The  explanation  appeared 
to  be  that  the  variability  in  the 
visual  thresholds  was  very  small, 
and  the  apparent  confidence  with 
which  the  observers  made  their 
judgments  was  very  high.  There 
was,  on  the  other  hand,  much  more 
variability  in  the  auditory 
thresholds,  and  the  observers  were 
much  more  hesitant  in  their 
judgments.  The  auditory  threshold, 
therefore,  was  open  to  improvement 
by  any  additional  visual  cues.  The 
confidence  with  which  the  visual 
judgments  were  made,  however,  left 
little  room  for  improvment,  and  the 
addition  of  auditory  cues  had 
little  effect. 

The  relative  variability  of  the 
auditory  thresholds  was,  however,  a 
weak  link  in  the  procedure. 

Although  it  was  possible  to  say 


Table  1 


Mean  Attenuation  at  Identification  Threshold  (dB) 


Visual  Auditory 


Target 

Base 

Uni 

Bim 

Base 

Uni 

Bim 

LM 

18.25 

18.15 

18.57 

57.22 

56.78 

59.35 

MH 

18.27 

17.70 

18.08 

56.00 

56.35 

59.69 

LH 

18.07 

18.35 

19.08 

57.85 

58.85 

60.22 

LMH 

17  59 

17.65 

18.38 

57.10 

57.55 

59.82 

Mean 

18. 0A 

17.96 

18.52 

57. 0A 

57.38 

59.77 
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with  considerable  confidence  what 
the  visual  threshold  was  for  a 
given  observer  and,  therefore,  to 
present  the  visual  signal  at  5  or  2 
dB  below  that  threshold,  the  same 
could  not  be  done  with  the  auditory 
thresholds.  We  could  not  be 
certain  that  we  were  a  given  number 
of  dB  below  that  threshold  and  that 
the  visual  and  auditory  signals 
being  presented  simultaneously  were 
perceptually  equivalent. 

Experiment  2  attempted  to  solve 
this  problem  by  presenting  the 
ancillary  signal  at  a  series  of 
constant  intensities  during  the 
determination  of  the  threshold  of 
interest . 

METHOD 

Subjects.  Four  volunteers  again 
participated,  three  staff  members 
and  one  sonarman.  One  of  the  staff 
members  had  participated  in 
Experiment  1;  the  others  were  new 
subjects. 

Apparatus.  The  same  apparatus  was 
used  as  in  Experiment  1. 

Procedure.  Each  session  again 
started  with  practice  in 
identifying  the  four  targets. 

Next,  the  baseline  thresholds  were 
measured  for  each  of  the  four 
targets  both  visually  and 
auditorially .  A  threshold  was  the 
mean  of  several  determinations.  If 
these  showed  little  variability, 
then  only  three  were  taken;  if  the 
variability  was  enough  to  cast 
doubt  on  the  threshold  value,  then 
as  many  as  10  determinations  were 
made . 

After  this,  the  visual  and 
auditory  thresholds  for  the  four 
targets  under  the  five  conditions 


were  measured  in  random  order.  The 
subject  uhus  did  not  know  whether 
the  threshold  being  measured  was 
visual  or  auditory,  unimodal  or 
bimodal.  In  this  experiment,  when 
a  threshold  *; as  being  measured  for 
one  modality,  the  stimulus  in  the 
other  modality  was  either  not 
present  or  was  presented  at  a 
constant  intensity  level  of  either 
1,  2,  3,  or  4  dB  below  the  baseline 
threshold  measured  at  the  start  of 
that  session. 

While  the  signal  in  the 
ancillary  modality  was  kept  at  a 
constant  intensity,  the  signal  in 
the  modality  whose  threshold  was 
being  measured  was  first  presented 
at  some  random  intensity  below 
threshold  and  increased  by  1  dB 
every  10  sec  until  the  subject 
correctly  identified  the  target. 

He  also  reported  whether  he  had 
heard  or  seen  the  target. 

The  threshold  in  each  condition 
was  measured  once  during  each 
session.  There  were  five  sessions 
for  each  subject,  resulting  in  five 
threshold  determinations  for  each 
condition.  A  session  lasted 
between  60  and  90  minutes. 

RESULTS 

Both  the  visual  and  auditory 
thresholds  were  measured  without  a 
stimulus  in  the  other  modality  as 
well  as  with  a  stimulus  presented 
from  1  to  4  dB  below  its  threshold 
measured  at  the  start  of  the 
session.  Table  2  gives  the 
differences  between  the  unimodal 
threshold  and  the  thresholds  in  the 
presence  of  stimulation  in  the 
other  modality. 

For  example,  the  mean  visual 
threshold  for  the  "LM"  target  was 
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Table  2.  Difference  in  attenuation  (dB)  at  threshold  between  the 
unimodal  and  bimodal  stimulation.  A  negative  number 
indicates  less  sensitivity  in  the  bimodal  condition. 
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Visual 


(Auditory  setting) 


Auditory 
(Visual  setting) 


Target 

-  ldB 

-2dB 

-3dB 

-4dB 

-  ldB 

-2dB 

-3dB 

-4dB 

| 

LM 

1.7 

1  .9 

0.8 

1  .  1 

1 .0 

1  .6 

0.5 

0.3 

1 

MH 

0.5 

0.8 

0.7 

0.2 

0.4 

-0.04 

0.4 

0.8 

a. 

£ 

LH 

0.5 

0.6 

0.2 

0.7 

0.8 

0.8 

1 .3 

0.9 

•v 

«*. 

LMH 

1.1 

1.2 

1 .0 

0.1 

1  .8 

1  .7 

2  .0 

1  .9 

1 

f.V 

* 

mm  * 

Mean 

0.95 

1  . 1 

0.7 

0.5 

1.0 

1  .0 

1 .0 

0.8 

b\' 

i 

N 

1 

28 

30 

45 

52 

103 

101 

100 

95 

V 

*/ 

► .  * 

p-value 

.02 

NS 

.01 

.01 

.001 

.001 

.002 

.002 

V 
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at  an  attenuation  of  17. l  dB.  In 
the  presence  of  the  auditory 
stimulus  presented  1  dB  below  the 
baseline  threshold,  the  subjects 
reported  seeing  that  target  at  an 
attenuation  of  18.8  dB,  an  increase 
of  1.7  dB. 

Table  2  shows  that  in  virtually 
every  condition,  the  subjects 
identified  the  targets,  either 
visual  or  auditory,  at  a  greater 
attenuation  —  that  is,  with 
increased  sensitivity  —  in  the 
presence  of  the  bimodal 
stimulation.  The  degree  of 
enhancement  declined  only  slightly 
as  the  level  of  the  ancillary 
stimulation  decreased.  The  mean 
differences  were  all  significant, 
according  to  the  Sign  Test  (Siegel, 
1956) ,  except  for  the  visual 
threshold  in  the  presence  of  the 
auditory  stimulus  2  dB  oelow  its 
threshold.  Once  again,  the 
enhancement  of  the  auditory 
thresholds  was  greater  than  that  of 
the  visual  thresholds. 

Table  2  also  shows  the  number 
of  thresholds  for  each  condition. 

It  is  obvious  that  more  auditory 
than  visual  thresholds  were 
obtained.  This  resulted  from  the 
fact  that  many  times  the  subjects 
reported  hearing  the  target  when  it 
was  kept  at  1  to  4  dB  below  the 
baseline  threshold  as  the  visual 
signal  was  being  increased.  This 
happened  must  less  often  with  the 
visual  thresholds.  It  was  these 
things  which  produced  the  unequal 
number  of  thresholds  in  the  various 
conditions.  Again,  this  supports 
the  statement  that  the  auditory 
thresholds  were  much  more  variable 
than  the  visual  thresholds. 


In  both  experiments,  the 
observers  identified  the  targets  at 
lower  intensity  levels  with 
redundant  bimodal  target 
information.  The  enhancement  was 
more  effective  for  the  auditory 
than  the  visual  stimuli.  This 
enhancement  was  fairly  consistent 
as  the  intensity  of  the  stimulation 
in  the  other  modality  decreased 
through  the  range  used  in 
Experiment  2,  particularly  for  the 
auditory  modality. 

When  the  target  intensity  is 
low,  the  observers  are,  of  course, 
unsure  as  to  the  identity  of  the 
target.  But  they  have  some  ideas. 
The  availability  of  additional 
information,  even  in  the  form  of 
stimuli  which  are  "below 
threshold",  improved  their  ability 
to  make  the  identification. 

The  reason,  of  course,  is  that 
the  threshold  is  not  a  definite 
point,  but  rather  a  somewhat 
indefinite  zone;  at  the  lower  end 
of  this  zone,  the  signal  may  be 
identified  only  10%  of  the  time;  at 
the  upper  end  of  the  zone,  it  may 
be  identified  90%  of  the  time. 

Thus,  although  we  set  a  signal  2  dB 
below  the  measured  threshold,  we 
must  expect  it  to  be  seen  a  certain 
proportion  of  the  time.  Moreover, 
the  zone  is  always  changing  from 
moment  to  moment.  Although  we 
established  a  baseline  threshold 
for  both  the  visual  and  auditory 
stimuli  at  the  beginning  of  each 
session,  this  undoubtedly  was 
constantly  changing  to  some  extent 
during  the  session.  When  we  set  a 
stimulus  1,  or  2 ,  or  4  dB  below 
that  threshold,  it  is  difficult  to 
know  if  it  was,  in  fact,  where  it 
was  supposed  to  be.  Thus,  there 
may  nave  been  times  when  a  stimulus 
set  at  -1  dB,  or  even  -4  dB,  was, 
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in  fact,  above  the  observer’s 
threshold  at  that  moment. 

But  there  can  be  no  question 
that  a  stimulus  at  -4  dB  was,  on 
average,  less  perceptible  than  one 
at  -3  or  -2  dB,  and  was  quite 
likely  below  threshold  a  good  part 
of  the  time.  Vet  identification  of 
these  stimuli  increased 
significantly  compared  to  when  they 
were  presented  unimodally.  It 
appears  that  the  observers  were 
using  not  just  the  best  information 
but  rather  were  using  all  the 
information  available  to  them  when 
attempting  to  identify  the  targets. 
Even  the  stimuli  which  were  below 
threshold  provided  some 
information.  Apparently,  even  a 
limited  amount  of  information  is 
useful  when  added  to  other 
information  in  another  modality. 

It  is  likely  that  what  is  happening 
is  this.  The  subject  may,  for 
example,  1 .  sure  that  he  hears  the 
high  and  ‘  frequencies  but  be 
uncertain  as  to  whether  or  not  he 
hears  the  middle  frequency.  The 
visual  display,  although  unclear, 
may  be  enough  to  convince  him  that 
the  middle  frequency  is  being 
presented. 

And  when  the  stimuli  are 
presented  for  extended  periods  of 
time,  fluctuations  in  threshold 
level  increase  the  probability  that 
the  target  will  be  above  threshold 
(or  closer  to  threshold)  for  some 
period  of  time.  One  would  expect 
that  the  less  the  secondary 
information,  the  less  the  threshold 
enhancement.  As  the  secondary 
stimulus  was  set  further  below 
threshold,  the  amount  of 
enhancement  was  reduced  suprisingly 
little.  This  suggests  that  the 
enhancement  is  not  simply  a  matter 
of  probability  but  rather  that 


observers  are  adept  at  extracting 
information  from  highly  degraded 
stimuli,  and  that  small  amounts  of 
information  enhance  the  confidence 
and  accuracy  of  target 
identification. 

If  the  auditory  information  can 
be  so  enhanced  by  processing  that 
it  is  brought  to  the  same  threshold 
level  as  is  now  done  for  the  visual 
information,  then  it  would  be  must 
more  to  the  advantage  of  the 
operator  to  have  both  presented 
simultaneously.  This  is  not  to 
say,  of  course,  that  auditory 
information  is  currently  useless; 
certain  signals  such  as  transients 
even  now  are  best  detected  by  ear. 
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